1. Field of the Invention
The invention relates generally to storage systems and more specifically relates to methods and storage systems providing improved performance in a declustered storage architecture by using back-end interface communication channels for N-way connectivity among all components of the storage system.
2. Discussion of Related Art
Storage systems have evolved from simple single disk drives to highly complex storage systems incorporating hundreds or thousands of disk drives and storage management techniques to improve both performance and reliability. In general, present day RAID (Redundant Array of Independent Disks) storage management techniques define logical storage volumes comprising multiple disk drives. Storage controllers associated with the system provide RAID storage management control logic to define and manipulate logical volume mapping host system I/O requests directed to a logical volume into lower level accesses to the underlying multiple physical disk drives. Attached host systems thereby perceive a single logical storage volume that has improved performance and reliability. Transparently to the host systems, data of the logical volume is distributed or striped over multiple disk drives to improve performance and redundancy information is added to the data stored in the logical volume to improve reliability (permitting continued operation without loss of data in response to failure of a disk drive in the logical volume). Larger RAID storage systems may define multiple such logical volumes defined as fixed portions of fixed groups of disk drives in the storage system.
Newer storage paradigms provide still further enhancements by distributing data over all disk drives of the entire storage system (e.g., “declustered” storage architecture). In one current embodiment of such a paradigm, the aggregate storage capacity of all storage devices in the system is treated as a pool of available physical storage and logical volumes defined by the RAID controller may be distributed in any useful manner over any of the pool of physical storage. Each logical volume is defined, in essence, by a mapping structure that identifies where blocks of data corresponding to logical block addresses of the logical volume are stored in the storage pool that is the physical disk drives of the system. These newer data distribution techniques may serve to provide, for example, faster recovery from drive failures, greater uniformity of performance across logical volumes, or lower power requirements. For example, a method known as Controlled Replication Under Scalable Hashing (CRUSH) may distribute data blocks of any single RAID level 5 stripe over any of the storage capacity of any of the storage devices of the system. CRUSH methods and structures utilize a hierarchical cluster map representing available storage devices in order to map logical to physical addresses and to permit migration of data all transparently with respect to attached host systems. CRUSH provides for a layer of virtualization above and beyond RAID logical volumes, wherein stored data may be migrated to any subset of the hundreds or even thousands of storage devices of the system. Furthermore, using CRUSH techniques, migration may occur as an online process, without interruption of processing of host I/O requests. In general, the storage controller in a storage system using the CRUSH architecture is coupled with all of the disk drives of the system to allow the controller complete flexibility to store and migrate physical storage anywhere it deems appropriate. Mapping features map all logical addresses and logical volumes to corresponding portions of physical storage. Other declustered, distributed storage management techniques are known to those of ordinary skill in the art where data is distributed over any of the storage devices of the storage system without regard to predefined, static groupings or clustering of the storage devices.
Another storage architectural feature applied in high-performance, high-reliability storage applications employs multiple storage controllers coupled with the pool of physical storage devices. Multiple storage controllers provide added reliability in that each may serve as a backup or redundant controller for the other controllers. Further, the multiple storage controllers may also be simultaneously active to process I/O requests from attached host systems and thus may provide enhanced performance.
Utilizing the CRUSH storage methods (e.g., declustered storage management as well as other complex mapped/virtualized storage architectures) with multiple storage controllers requires that each of the multiple storage controllers be capable of communicating with each of the storage devices in the system (i.e., with each of the disk drives managed in accordance with the CRUSH storage management architecture). An architecture in which each of multiple storage controllers of the system has some form of access to each of the plurality of storage devices is sometimes referred to as “N-way” connectivity or “any-to-any” connectivity.
As presently practiced, each storage controller in a CRUSH architecture, multi-controller storage system is coupled with some subset of locally accessible disk drives through its “back-end” interface and is coupled with the other storage controllers (inter-controller connection) through some other interface of each storage controller. Since N-way connectivity is required for CRUSH storage management (as well as other complex storage techniques), the inter-controller connectivity is used in present systems to access other disk drives of the storage system that are not locally accessible to that controller (i.e., N-way connectivity is achieved through the inter-controller communications capabilities).
In some exemplary embodiments, the “front-end” interface of each storage controller may be used to communicate with attached host system and may also serve as the inter-controller communication path to enable communication with all storage devices through other controllers. Such embodiments suffer from poor performance in that the bandwidth of the communication media coupled with the front-end interface of each controller may be over-utilized. Where the front-end interface couples the storage controller to a common network also used for host system inter-connections, that “network” medium may be overutilized by adding the storage system inter-controller communications.
In other embodiments, a dedicated communication medium may be used for inter-controller communications between the storage controllers. For example, a PCI Express fabric, an Infiniband fabric, or a Fibre Channel fabric may be employed as a dedicated inter-controller communication medium for the multiple storage controllers. Such solutions add complexity (and hence cost and additional potential points for failure) to the storage system architecture. The added complexity creates problems for scaling the storage system to different application needs
Thus it is an ongoing challenge to provide N-way connectivity in a storage system to permit all storage controllers of the system to access all storage devices of the system.